Video/Audio Transcription and Summarization

Preface

This On-the-Job Training (OJT) project provided an invaluable opportunity to apply theoretical knowledge of natural language processing, deep learning, and data engineering into a real-world application. The primary objective was to build a pipeline for automatic transcription and summarization of educational and conference video content, aimed at making such content more accessible and searchable.

I would like to express my sincere gratitude to Dr. Dinesh Helwade for his guidance and encouragement throughout the project. It was under his initiative that this project was designed with a vision to open-source transcription and summarization tools for the benefit of educational institutions and research communities.

This project marks a critical step in combining machine learning technology with accessibility goals in education.

Introduction/Problem Statement

Video content has become the most preferred medium for disseminating information — especially in academia, conferences, and online education. However, extracting insights or reviewing content from long-form videos remains time-consuming and inefficient.

In recent years, video content has emerged as the dominant form of communication, especially in education, conferences, and online platforms. With affordable smartphones and high-speed internet, everyone can now record and upload lectures, presentations, or tutorials. But while we have tons of valuable information in videos, extracting insights from them is still a challenge. Videos are inherently unstructured — there’s no easy way to search through them, skim key points, or even understand the content without watching them in full. That’s where automatic transcription and summarization becomes vital — not just for accessibility, but also for efficient knowledge retrieval and documentation. Our project aims to solve this by building a complete, end-to-end pipeline that takes in a video and outputs structured, summarized text.

Key Challenges

Manual note-taking is tedious and error-prone
Searchability within videos is limited
Lack of concise summaries for quick reference
Existing transcription tools are often paid or closed-source

Let’s dive into the specific problem we’re addressing. First, transcription and summarization are essential but tedious tasks if done manually — especially for long lectures or talks. They require time, accuracy, and domain understanding — something that’s hard to scale. Secondly, educational content often includes complex terminology, equations, and multi-speaker interactions, making accurate speech recognition even harder. Third, many existing transcription tools are cloud-based, expensive, or require constant internet — not ideal for academic or institutional use. What we needed was a flexible, GPU-accelerated, open-source pipeline that can be used even offline, with support for different models, good accuracy, and ease of deployment. That was the challenge — and the motivation behind this project.

About Organization

Pradnyaa InfoVision, headquartered in Pune, India, is a specialized analytics and consulting firm with over four years of experience delivering data-driven solutions. The company operates across two core domains: Retail Analytics and Biostatistics, offering deep domain expertise and tailored consulting to global clients.

🛒 Retail Analytics Division

The Retail Analytics team at Pradnyaa InfoVision excels in demand forecasting, leveraging both standard statistical models and cutting-edge machine learning algorithms. Their capabilities span across:

Regular Price Optimization
Markdown Strategy & Optimization
Inventory Optimization

One of the firm’s flagship projects involved partnering with a major U.S. retailer to design and execute a comprehensive price test across the entire U.S. region—demonstrating the company’s global reach and strategic insight.

🧪 Biostatistics & Clinical Trial Analytics

In the life sciences space, Pradnyaa InfoVision plays a key role in the analysis and processing of all three phases of clinical trials. The Biostatistics division supports pharmaceutical and healthcare companies by providing end-to-end statistical solutions that comply with global regulatory standards.

🌍 Offshore Delivery and Managed Services

Beyond analytics, the company also provides strategic consulting for offshore office setup and management in India. This includes team recruitment, operational oversight, and seamless integration with client business processes, enabling clients to establish a strong and scalable presence in India.

Project Objectives

Automate Speech-to-Text Conversion
Implement Robust Summarization
Design an Interactive Web Interface
Support Long Audio/Video Files
Ensure Local and Scalable Deployment
Improve Accessibility and Productivity

The project was driven by six main objectives:

First, to automate speech-to-text conversion using advanced models like Whisper. This automation drastically reduces the manual labor involved in transcribing lectures or meetings.

Second, to implement robust summarization techniques so users can quickly grasp the essence of lengthy content. We experimented with both traditional and transformer-based summarizers for this.

Third, to design an interactive web interface that simplifies the workflow. Whether it’s uploading files, selecting models, or downloading outputs, the interface was made intuitive and user-centric.

Fourth, to support long audio or video files, which is essential for real-world educational and conference materials that often span over an hour.

Fifth, we aimed to ensure the system could be deployed locally, especially for institutions without reliable internet. At the same time, it needed to be scalable if deployed on cloud infrastructure.

And finally, we wanted the tool to enhance accessibility and productivity — by making academic and technical content more searchable, reviewable, and shareable.

Pipeline

This slide outlines the core pipeline that was developed during the OJT.

Step 1: The user begins by uploading an audio or video file through the web interface. We made sure the app accepts common formats like .mp4, .mp3, .wav, etc.

Step 2: If a video is uploaded, the system extracts the audio using FFmpeg and then resamples it to 16kHz mono WAV format — which is the requirement for the transcription models.

Step 3: The audio is transcribed using either Whisper or its faster variant, Faster-Whisper. The model is loaded lazily only after the user selects it, optimizing memory usage.

Step 4: The raw transcript is cleaned and broken into manageable chunks. This ensures better summarization and avoids truncation during model inference.

Step 5: The processed chunks are summarized using the facebook/bart-large-cnn model. This step significantly reduces the size of the text while retaining the core message.

Step 6: Finally, the output is shown in the UI and the user is given options to download the transcript and summary in .txt format.

Tools & Technologies Used

Speech Models: Whisper(Base & Medium), Faster-Whisper
Summarization: BART (facebook/bart-large-cnn), SUMY(Luhn)
Web Interface: Streamlit
Preprocessing: FFmpeg
Programming Language: Python
Others: ONNX Runtime, Transformers

In this slide, we highlight the major tools and frameworks used in the project.

Starting with the transcription models, we used Whisper and Faster-Whisper — both known for their accuracy and ability to handle multilingual input. Faster-Whisper offers better speed and efficiency due to its optimized inference engine.

For summarization, we used BART, specifically the facebook/bart-large-cnn model, which is well-suited for document-level abstractive summarization.

On the web interface side, the final implementation uses Streamlit because of its lightweight design and ease of development. We also explored Shiny for Python for comparison, but ultimately preferred Streamlit for its simplicity and deployment flexibility.

FFmpeg was used to handle audio extraction and conversion from video files, and waveform processing.

The entire project was built in Python, taking advantage of modern NLP libraries like Transformers and ONNX Runtime for efficient model execution.

All these tools together enabled us to create a modular, scalable, and responsive speech-to-text summarization system.

Model Comparison: Features at a Glance

Model	Accuracy	Offline	Multilingual	Ease of Use	Cost
Whisper	✅✅✅	✅	✅✅✅	✅✅	Free
Wav2Vec 2.0	✅✅	✅	⚠️ (mostly English)	✅	Free
Google API	✅✅✅	❌	✅✅✅	✅✅✅	Paid
DeepSpeech	✅	✅	❌	✅✅	Free
Kaldi	✅✅	✅	✅ (with effort)	⚠️ Complex	Free
Vosk	✅✅	✅	✅✅	✅✅✅	Free

This table summarizes the strengths and limitations of some of the most widely used speech-to-text models and toolkits.

Starting with Whisper, it stands out as a balanced option — it offers high accuracy, works offline, supports multiple languages, and is open-source. It does require a decent GPU to run efficiently, especially for long audio files or real-time use, but it’s very accessible for developers and teams with modest resources.

Wav2Vec 2.0 is another excellent option. It provides high accuracy, especially when fine-tuned on domain-specific data. However, it’s primarily trained on English datasets, so multilingual support is more limited unless you retrain it. It’s open-source and can be accessed via Hugging Face.

Google Speech-to-Text API is a commercial solution. It offers very high accuracy and supports over 100 languages. It’s excellent for real-time use cases and scales easily. However, it’s not free — the cost is around $1.44 per hour, and it requires internet connectivity, which may raise privacy concerns in sensitive environments.

DeepSpeech was designed to be lightweight and fast, making it suitable for edge devices. That said, it has started to lag behind in terms of accuracy, especially on noisy or accented inputs, and its development has slowed down in recent years.

Kaldi is extremely powerful and flexible. It’s used in many academic and enterprise applications where low-level control is needed — for instance, custom language models, speaker adaptation, and complex pipelines. However, it’s not beginner-friendly, and the learning curve is steep. PyKaldi can help if you want to work with it in Python.

Vosk is surprisingly effective given its small size. It offers decent accuracy, real-time performance, works fully offline, and is extremely easy to integrate — especially on embedded systems or mobile platforms.For projects targeting Indian languages or edge devices, Vosk can be a better fit than Whisper or cloud APIs.

This slide helps us decide which model might be most appropriate depending on our project constraints — whether we prioritize cost, offline access, multilingual support, or ease of use.

In the next slide, we’ll break down the cost aspect more specifically, including free tiers and usage pricing for cloud APIs.

Limitations

Every machine learning project has constraints.
This section highlights key model, summarization, and deployment limitations.
Helps define boundaries for interpretation and guides future improvements.

Model-Level Limitations: Whisper

Hardware Requirements: Large Whisper models need GPUs; CPU inference is slow.
Inconsistent Formatting: Output lacks punctuation and proper sentence segmentation.
Limited Domain Understanding: Struggles with mathematical or technical language.
Accent and Noise Sensitivity: Accuracy drops slightly with strong accents or noise.

Model-Level Limitations: BART

Hallucinations: May invent information not in the transcript.
Chunking Issues: Large inputs must be split, hurting context retention.
Factual Inaccuracy: Sometimes combines unrelated ideas.
Weak with Technical Content: Fails to handle formulas or structured arguments.

Project-Level Limitations

English-Only Evaluation: Indian languages like Hindi, Marathi not tested.
Slow Inference: Large models cause delays even with GPU.
No Real-Time Support: Pipeline is batch-based, not live.
Minimal Post-Editing: Output not cleaned for publication.

Future Scope

Fine-tune models for domain-specific use (math, stats, etc.)
Add punctuation restoration and sentence boundary detection
Evaluate on regional languages and accents
Explore lightweight or multilingual alternatives (e.g., DistilBART)
Research math-aware or rule-based summarizers

Key Learnings

Gained hands-on experience with speech-to-text pipelines
Understood limitations of transformer-based summarizers
Built a real-world app using Streamlit and ONNX
Improved skills in modular Python design and lazy loading
Learned how to evaluate model fit for domain-specific data

App

This project provided an opportunity to combine theoretical understanding with practical implementation. Working with Whisper and BART helped deepen my knowledge of transformer models and their real-world behavior.

Developing the app with Streamlit required careful architectural planning — from lazy loading large models only when needed, to handling long audio files efficiently. I also learned how to evaluate different tools not just based on performance but also deployment feasibility — for instance, comparing GPU vs CPU inference or assessing ONNX integration for faster execution.

The most valuable takeaway was understanding that building an end-to-end ML pipeline involves more than just good models — you need UX design, robustness, and awareness of hardware and domain limitations.

Conclusion

We successfully built an end-to-end speech-to-text and summarization pipeline.
Leveraged powerful models like Whisper and BART for high-quality transcription and summarization.
Explored alternative models and toolkits for varied use cases (offline, multilingual, edge devices).
Deployed the solution using Streamlit to provide a user-friendly interface.
Identified limitations with respect to summarizing technical or mathematical content.
Gained practical experience with NLP tools, model selection, and pipeline integration.

Acknowledgements

Dr. Dinesh Helwade for guidance and support
Purva Puranik and Pranav Ransing for technical assistance
Pradnyaa InfoVision for providing the platform and resources
Open-source community for tools like Whisper, BART, and Streamlit

Thank You!

I appreciate your time and attention.
Looking forward to your questions and feedback.

Scan the QR code to view the presentation: